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Abstract. Recently Verdu and Weissman introduced erasure entropies, which are meant 
to measure the information carried by one or more symbols given all of the remaining 
symbols in the realization of a random process or field. A natural relation to Gibbs 
measures has also been observed. In this short note we study this relation further, review 
a few earlier contributions from statistical mechanics, and provide the formula for the 
erasure entropy of a Gibbs measure in terms of the corresponding potential. For some 2- 
dimensional Ising models, for which Verdii and Weissman suggested a numerical proedure, 
we show how to obtain an exact formula for the erasure entropy. 



1. Introduction 

1.1. Erasure entropies. Recently, Verdu and Weisman [2I1I22], motivated by questions 
from information theory, introduced and studied erasure entropies. In one dimension the 
description is as follows. Let ^ = {Xn}n£i be a stationary process, taking values in a 
finite alphabet A. The erasure entropy of a collection of random variables {Xi, . . . , X„} is 
defined as 



H (Xi, . . . , Xn) = H{Xi\X\i), 



i=l 

where X\i = {Xj : j = 1, . . . ,n, j i}, and H(Xj|X\j) = H(Xi, . . . , Xn) — H(X\j) is the 
conditional entropy. 

Definition 1 (|2T]). The erasure entropy rate of the process ^ = {Xn}nez is defined 
as 

(2) h"(jr)= lim -H-(Xi,...,X„). 

The erasure entropy measures the information required to recover an erased symbol, 
knowing all the other symbols. 

Verdii and Weissman established the following properties of erasure entropies. 

Theorem 2 ([21]). The limit in ([2]) exists. Moreover, 

(3) h-(^)= lim H(Xo|X_„,...,X_i,Xi,...,X„), 

and 

(4) h"( JT) < h( JT) = lim H(Xo|X_i,...,X_„). 

n— >oo 
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If ^ = {Xn}n£Z is a k-step Markov chain then 

^^^-^ _ h + H( Xi, ■ ■ ■ , Xk\X_i, . . . ,X_ 



k + 1 

The interest in objects like the erasure entropy ([1]) and the corresponding entropy rate 
(E]) arose in information theory in connection to coding problems for discrete memoryless 
erasure channels (DME). 

Suppose that ^ = {Xi} is a discrete stationary process with values in a finite alphabet 
= {!,..., A^}, and consider the process ^ = {Zi} with values in alphabet ^ U {*}, 
defined for all i by 

'Xi, ifEi = 0, 
, if Ei = 1, 

where {Ei} is a sequence of Bernoulli random variables with 

P(^, = l)=p. 

The process {Ei} is also assumed to be independent of The resulting process ^ is 
the response of the DME with parameter p over The information rate at which the 
observer of 5° has to be supplied in order to reconstruct the erased symbols almost surely 
is given by 

h(jr|J^)= lim -H(Xi,...,X„|Zi,...,Z„) 
and, generically, is very difficult to evaluate. In [22], it was shown that 

h(jr|^) >ph-(^) VpG[0,l), 

and 

h(^|J^) =ph"(er) + o(p) asp^O. 

2. Bilaterally deterministic processes: 
source of counterexamples 

A stationary process = {Xn}nez is called deterministic if Xq is measurable with 
respect to the cr-algebra 23(Xfc, A; < 0). By stationarity this means that for every G N 

k<-n)=-B{..., Xo, X,,...), 

and hence Xq can be predicted with zero error probability from its past (. . . , X_2, -'^-i)- 

Analogously, a stationary process ^ = {Xn}nez is called bilaterally deterministic 
if for every n G N, 

S(Xfc,|A;|<n) = S(...,X„i,Xo,Xi,...), 
which is equivalent to requiring that given the past {. . . , X_n-i, X^^} and the future 
{X„,X„_|_i, . . .}, the value Xq can be reconstructed with zero error probability. Therefore, 
any bilaterally deterministic process has erasure entropy rate 0. 

Gurevic [6] constructed a first example of a non-deterministic, but bilaterally determin- 
istic, stochastic process; for a strongly mixing example see [4J. Any deterministic process 
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is clearly bilaterally deterministic. The converse is false in quite a strong sense, as the 
following somewhat counter-intuitive result obtained by Ornstein and Weiss [13] shows. 

Theorem 3. Given any ergodic finite-state stationary stochastic process, there is an iso- 
morphic ergodic finite-state stationary stochastic process which is bilaterally deterministic. 

Isomorphism here is understood in measure-theoretical sense. More specifically, in [H] 
a finite generating partition /? = {Bi, . . . , Bm} of is built such that the corresponding 
factor process V = {Yn} given by 

Yn = j (T"({Xfc}fc6z) G Bj for all n, 

where a : — )■ A^ is the left shift, is bilaterally deterministic. The fact that the partition 
P is generating means that given {1^} one can reconstruct almost surely. 

Theorem [3] implies that the erasure entropy rate h~(^) is not a measure-theoretic 
invariant of the process ^ = while the entropy rate h(^) is well known to be such 

an invariant. 



3. Erasure entropies of Gibbs states 

A natural relation between erasure entropies and thermodynamic (Gibbs) states was 
observed in the original paper [5T]. In particular, the application to Gibbs sampling of 
Markov random fields, a well-known problem in stochastic image analysis, served as a mo- 
tivation for considering this question also on higher-dimensional lattices, see in particular 
their section 4F, in which they provide an approximate expression for the erasure entropy 
of a two-dimensional nearest-neighbour Ising model. Let us elaborate on this issue a bit 
further. 

3.1. Gibbs measures. Gibbs measures for a given interaction $, according to the defini- 
tion introduced by Dobrushin, Lanford and Ruelle, are defined as measures whose condi- 
tional probabilities of finite- volume configurations X\, given external configurations X\c, 
are of the the Gibbsian form 

F{Xa\Xm) = -^exp(-Ht{X)) =: 7*(XA|XAe) 

where the Hamiltonian if* contains all interaction terms having a non-empty intersection 
with A: 

HtiX)= "^^W' 

where the normalization constant -^x^c appropriate partition function. 

This should hold for all volumes A C Z'^, and all choices of the configurations Xa and X^c 

See e.g. 
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3.2. Erasure entropy rates of random fields. For a finite volume A C Z'' and any 

k G Z'^, we describe the shifted volume k + A = {k + i|i G A}. 

Definition 4. Let {X„}„g^d be a translation-invariant random field taking values in a 
finite alphabet A, and let P be the corresponding probability measure on A^. The erasure 
entropy rate of P for the finite volume A C Z"' zs defined as 

(6) hX (P) = lim ^ 5^ H (X(k+A)nv |^y\(k+A)) 

The interpretation of the erasure entropy for a finite volume remains unchanged: hX(P) 
is the number of bits required to reconstruct the configuration on A given the configuration 
outside of A. In fact, it is also convenient to consider the erasure entropy normalized by 
the volume ■j^hX(P). 

Theorem 5. The limit ^ exists, and 

(7) h;,(P) = lim H{Xa\Xv\a) = - [ logP(XA|XAc)P((iX). 

V J 

For Gibbs measures one can say more. 

Theorem 6. If F is a translation invariant Gibbs measure on A^'' for interaction $, then 
the erasure entropy rate can be expressed in terms of the Gibbs specification 7* as follows: 



Hence, h^(-) is an affine functional on the set of Gibbs measures consistent with specifica- 
tion 7*. Moreover, 

(9) ^lta±hX(P) = h(P). 

Proof. Proofs of such results are rather standard and rely on application of (strong) sub- 
additivity, martingale convergence and ergodic theorems, [Tl[l9l|2T]. Expression (|8]) is an 
immediate consequence of the Gibbs property of P. To show ([9]) we note that 

= j l-^Ht{X)¥{dX) + I i^logZ(XAc)P(dX)^h(P) 

by in Corollary 15.35]. □ 

Let us conclude this section with a number of remarks on the relation between erasure 
entropies and the theory of Gibbs states of Equilibrium Statistical Mechanics. 

(A) By ([7]), the erasure entropy rate can also be seen as the difference in entropies of 
the system with and without a finite number of spins inside an interval or a volume. As 



ERASURE ENTROPIES AND GIBBS MEASURES 



5 



such, erasure entropy is an example of an inner entropy (in some literature called local or 
conditional entropy ) introduced earlier in Statistical Mechanics, see [IHIIIS] or [T^[2U]. 

Such conditional entropies have been used before to establish "local" or "inner" varia- 
tional principles [5l[T6l[I71ll9l[20]. From ([9]) it is known that for Gibbs measures the limit 
conditional entropy density and the standard entropy density coincide [71|20]. This prop- 
erty is obviously not true in general, due to the Gurevic and Ornstein- Weiss examples, see 



(B) One extra remark to be made is that Gibbs measures can be characterised, not only 
by a local variational principle for general volumes, but even by a single-site variational 
principle, using only the (Verdii-Weissman) single-site erasure entropy. This was proven in 
a short paper by J. Fernando Perez and R.H. Schonmann [9j. More specifically, suppose 
$ = {$a} is an interaction and P is an arbitrary probability measure (state). The free- 
energy content of a region A in a state P is defined by 



Here we do not assume P to be translation invariant, and hence the erasure entropy hA(P) 
is understood as the limit 



Following Sewell [T7], a state P is called locally thermodynamically stable if for every 
finite volume A one has 



for all measures P such that Pa^ = Pa^. In other words, the locally thermodynamically 
stable state minimizes the free-energy content with respect to variations of the state inside 
A, for all A. 

It turns out that for a fairly large class of finite-norm interactions, from the fact that 
P minimizes Fa, one can deduce that P satisfies the Dobrushin-Lanford-Ruelle (DLR) 
conditions on A, see [U Theorem 2]. Moreover, it also known that if a translation- invariant 
state P satisfies DLR condition on A = {0}, then P is a Gibbs state. (Or, more generally, 
if the DLR conditions hold for all single sites they hold in general). 

Hence, we can formulate a variational principle for Gibbs states in terms of the erasure 
entropy hg (P). 

Although [9] formulate their result for finite-range interactions, it is easy to see that 
their arguments generalise beyond that, due to the analysis in [51IIS] of the LTS condition 
and its equivalence to other equilibrium conditions. 



e.g. ^m\. 




hA(P) = lim H(Xa|X^\a). 



Fa(P) > Fa(P) 



(C) It follows from the bilaterally deterministic examples discussed in Section 2 that 
neither the erasure entropy, nor the Gibbs property are measure-theoretical invariants, c.f. 
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4. Computing erasure entropies for Ising models in two dimensions 

In [22] a question arose about computation of a 1-site (i.e., A = {0}) erasure entropy 
rate for the standard nearest-neighbour Ising model on the two-dimensional integer lattice. 
In fact, in contrast to what was suggested there, this erasure entropy rate can be computed 
explicitly. 

Because of the Markov property of the Gibbs measures of the 2-dimensional Ising model 
(immediate from the nearest- neighbour character of the interaction), it is sufficient to 
consider nearest-neighbour environments of the spin at the center. Let us denote the 
5 spins we will need to consider ac,crE,<JN,o'w,o's (for Center, East, North, West, and 
South). 





o"v 




aw 








0-5 





Figure 1. Ising spins 

We first remark that the probabilities of the 16 configurations of the E., N., W., and S. 
spins, in the spin-flip-invariant inflnite-volume Gibbs measure P = l(p+ + p-), fall into 4 
classes. 
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Figure 2. Four boundary types (upto rotations and spin-flips) 

Denote by Pj, z = 1, . . . , 4 the probability of the corresponding boundary conflguration. 
The appropriate combination of these 4 probabilities adds up to 1: 

(10) 2Pi + 8P2 + 2P3 + 4P4 = 1. 

By (jHD, the erasure entropy h|Q|(P) is given by: 

+ (2P3 + 4P4).ft(i). 

where h{p) = — plogjo — (1 — p)log(l — p) for p G [0, 1], and J is the nearest-neighbour 
coupling constant. Note that in [22j the third and the fourth boundary types are not 
distinguished (since they contribute equally to the entropy). However, to actually compute 
the erasure entropy h^g|(P) we must distinguish these types. 

By Pfaffian methods (albeit in somewhat implicit form, see [I2]), one can compute 
three relevant correlation functions, namely the 4-point function and the two pair cor- 
relation functions one can build with these four spins, that is K{aEcrw),^{o'EO'N), and 
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E,{aEcrN<^wCs)- Finally, we derive remaining equations for probabilities Pi, by noting that 

(11) EiaECTNcrwcrs) = 2P, - 8P2 + 2P^ + AP^, 

(12) E{aEaw) = 2Fi + 2P3 - 4F4, 

(13) E(asa^) = 2Fi - 2P3- 
As an immediate consequence, one gets 

1 + E{aE(TN<^W(^s) + 2E{aECrw) + 4E((7£;CrAr) 



Pi 



16 

E{aEaN(^w(^s) 



16 

which is sufficient to express the erasure entropy in terms of correlation functions. 
In conclusion, we note that for obvious symmetry reasons 

h^o^(p-) = h^o^(P+), 

and, by the affine property, the erasure entropies of both phases coincide with h^Qi,(P) 
obtained above. For the same reasons, the erasure entropy of any Gibbs measure of the 
Ising model coincides with h^Q|(P). 

Definitions of erasure entropies extend readily to more general lattices. For the corre- 
spoding nearest-neighbour Ising models the Pfaffian method of computing correlations will 
apply to some other two-dimensional planar lattices, see again [IIE1II2] and specifically [2]. 
In case of the hexagonal lattice, we propose another approach, which is less general, in the 
sense that everything we need can be derived from just knowing the free energy density, 
but already leads to an explicit analytic expression of the erasure entropy. 

Consider the two-dimensional hexagonal (honeycomb) lattice. One now needs to consider 
only three neighbours - denoted aj, an, and am, of the central spin ac- There are 
obviously only 2 boundary types. 








Figure 3. Ising spins on hexagonal lattice and two boundary types. 



Consider again the zero-field situation and the symmetric Gibbs measure P. Let Pi, 
i = 1,2, denote the probability of the corresponding boundary type. Thus we have 

(14) 2Pi + 6P2 = 1. 
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For each quadruple (ct/, ct//, ct///, ac) € {—1, ^}^, one has 

P(a/, an, am, ac) = F{ac\crj, a^, ajjj)F{aj, an, am) 



p.. 



2 cosh {J{ai + an + ani)) 

where the boundary type i is determined by (a/, o"//, am). 
Consider now the correlation function E(cr/crc), 

^{aiac)= ^ aiacF{ai,an,crni,crc), 

{o-/,o-/j,(7/j/,(7c)G{-1,1}'' 

and after a straightforward manipulation we conclude that 

E{aiac) = Pitanh(3J) + F2tanh(J). 

Hence, 

6E(cT,cTc) - tanh( J) _ tanh(3 J) - 2E{aiac) 

Uoj -Ti — — — r7o~n oT — TTTT ' -'2 



6 tanh(3 J) - 2 tanh( J) ' 6 tanh(3 J) - 2 tanh( J) 

and the erasure entropy hQ"(P) 

3J J 

(16) h„-(P) = 2P,.A(-3^)+6P..ft(-^) 

can be expressed in terms of the correlation function K{aiac)- 

In order to obtain the expression for E(cr/0"c) we recall that the zero-field Ising model 
on the hexagonal lattice is an exactly solvable model, and the pressure function at inverse 
temperature f3 is given by 

-.277 /.27r 



p{P) = ^ log2 + -—r / / log(cosh=^(2/3J) + 1 - sinh2(2/3J)x 



cos(6'i - 62) + cos(6'i) + cos(6'2) )d0id02. 



3 

(17) 4 



see e.g. [H], eq. (A. 6). p. 322]. The pressure function p{(3) is an everywhere differentiable 
function of /?; the phase transition in the Ising model on hexagonal lattice is of second 
order. 

Finally, as is well known pT| [T^. Gibbs states are tangent functionals to the pressure, 
and hence, if the interaction $ = {$a} is such that the pressure function p{f3) = P{j3^) is 
differentiable function of /3 at /3 = /3o, then 



Ago ' ' 



where P is a Gibbs state for the Hamiltonian j3H = (3 $a- In case of the Ising model 
on the hexagonal lattice, we thus conclude that 

3 1 

(19) p'(l) = _:^E(a,ac). 
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Combining ( IT9l) . (|T71) . ( IT5l) . and (|T6l) . we arrive to an analytic expression for (P). Simi- 
larly to the case of rectangular lattice, the erasure entropy is independent of the phase. 

Final remarks A possible generalisation rests on the fact that the coupling constants 
in the three different directions are allowed to be different, without affecting the solubility 
of the model. In that case the corresponding correlation functions need to be separately 
computed. 

Also, the next nearest-neighbour correlation functions E((T/(T//) etc, can be computed 
as the nearest neighbour correlation functions of an Ising model on a triangular lattice, 
related to the hexagonal model by a star-triangle transformation, see again [18j. This offers 
a slightly different, though essentially equivalent, route to computing exact expressions for 
the Verdii-Weissman erasure entropy 
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